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Abstract 

Suppose we would like to know all answers to a set of statistical queries C on a data set up to small 
error, but we can only access the data itself using statistical queries. A trivial solution is to exhaustively 
ask all queries in C. Can we do any better? 

1. We show that the number of statistical queries necessary and sufficient for this task is — up to 
polynomial factors — equal to the agnostic learning complexity of C in Kearns' statistical query 
(SQ) model. This gives a complete answer to the question when running time is not a concern. 

2. We then show that the problem can be solved efficiently (allowing arbitrary error on a small fraction 
of queries) whenever the answers to C can be described by a submodular function. This includes 
many natural concept classes, such as graph cuts and Boolean disjunctions and conjunctions. 

While interesting from a learning theoretic point of view, our main applications are in privacy- 
preserving data analysis: Here, our second result leads to an algorithm that efficiently releases differ- 
entially private answers to all Boolean conjunctions with 1% average error. This presents significant 
progress on a key open problem in privacy-preserving data analysis. Our first result on the other hand 
gives unconditional lower bounds on any differentially private algorithm that admits a (potentially non- 
privacy-preserving) implementation using only statistical queries. Not only our algorithms, but also most 
known private algorithms can be implemented using only statistical queries, and hence are constrained 
by these lower bounds. Our result therefore isolates the complexity of agnostic learning in the SQ-model 
as a new barrier in the design of differentially private algorithms. 
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1 Introduction 



Consider a data set D c {0, \} d in which each element corresponds to an individual's record over d binary 
attributes. The goal of privacy-preserving data analysis is to enable rich statistical analyses on the data set 
while respecting individual privacy. In paritcular, we would like to guarantee differential privacy [DMNS06], 
a rigorous notion of privacy that guarantees the outcome of a statistical analysis is nearly indistinguishable 
on any two data sets that differ only in a single individual's data. 

One of the most important classes of statistical queries on the data set are Boolean conjunctions, some- 
times called contingency tables or marginal queries. See, for example, [BCD + 07, BLR08, KRSU10, UV10]. 
A boolean conjunction corresponding to a subset S c [d] counts what fraction of the individuals have each 
attribute in 5 set to 1. A major open problem in privacy -preserving data analysis is to efficiently create a 
differentially private synopsis of the data set that accurately encodes answers to all Boolean conjunctions. 
In this work we give an algorithm with runtime polynomial in d, which outputs a differentially private data 
structure that represents all boolean conjunctions up to an average error of 1%. 

Our result is significantly more general and applies to any collection of queries that can be described 
by a low sensitivity submodular function. Submodularity is a property that often arises in data analysis and 
machine learning problems [KG07], including problems for which privacy is a first-order design constraint 1 . 
Imagine, for example, a social network on d vertices. A data analyst may wish to analyze the size of the 
cuts induced by various subsets of the vertices. Here, our result provides a data structure that represents all 
cuts up to a small average error. Another important example of submodularity is the set-coverage function, 
which given a set system over elements in some universe U, represents the number of elements that are 
covered by the union of any collection of the sets. 

The size of our data structure grows exponentially in the inverse error desired, and hence we can rep- 
resent submodular functions only up to constant error if we want polynomial query complexity. Can any 
efficient algorithm do even better? We give evidence that in order to do better, fundamentally new tech- 
niques are needed. Specifically, we show that no polynomial-time algorithm that guarantees small error for 
every boolean conjunction can do substantially better if the algorithm permits an implementation that only 
accesses the database through statistical queries. This statement holds regardless of whether such an imple- 
mentation is privacy-preserving. (A statistical query is given by a function q: {0, \} d — > {0, 1}, to which the 
answer is E v£ £)[g(;t)].) 

We show this limitation using connection between the data release problem and standard problems in 
learning theory. Putting aside privacy concerns, we pose the following question: How many statistical 
queries to a data set are necessary and sufficent in order to approximately answer all queries in a class C ? 
We show that the number of statistical queries necessary and sufficient for this task is, up to a factor of 0(d), 
equal to the agnostic learning complexity of C (over arbitrary distributions) in Kearns' statistical query 
(SQ) model [Kea98]. Using an SQ lower bound for agnostically learning monotone conjunctions shown 
by Feldman [FellO], this connection implies that no polynomial-time algorithm operating in the SQ-model 
can release even monotone conjunctions to subconstant error. Since monotone conjunction queries can be 
described by a submodular function, the lower bound applies to releasing submodular functions as well. 

While the characterization above is independent of privacy concerns, it has two immediate implications 
for private data release: 

• Firstly, it also characterizes what can be released in the local privacy model of Kasiviswanathan et 
al. [KLN + 08]; this follows from the fact that [KLN + 08] showed that SQ algorithms are precisely what 
can be computed in the local privacy model. 

'For example, Kempe, Kleinberg, and Tardos show that for two common models of influence propagation on social networks, 
the function capturing the "influence" of a set of users (perhaps the targets of a viral marketing campaign) is a monotone submodular 
function [KKT03]. 
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• Secondly, and perhaps even more importantly, it gives us the claimed unconditional lower bounds on 
the running time of any query-release algorithm that permits an implementation using only statisti- 
cal queries — regardless of whether its privacy analysis can be carried out in the local privacy model. 
To our knowledge, this class includes almost all privacy preserving algorithms developed to date, in- 
cluding the recently introduced Median Mechanism [RR10] and Multiplicative Weights Mechanism 
[HR10] 2 . Note that these mechanisms cannot be implemented in the local privacy model while pre- 
serving their privacy guarantees, because they will have to make too many queries. Indeed, they are 
capable of releasing conjunctions to subconstant error! Yet, they can be implemented using only 
statistical queries, and so our lower bounds apply to their running time. 

To summarize, our results imply that if we want to develop efficient algorithms to solve the query release 
problem for classes as expressive as monotone conjunctions (itself an extremely simple class!), we need to 
develop techniques that are able to sidestep this statistical query barrier. On a conceptual level, our results 
present new reductions from problems in differential privacy to problems in learning theory. 

1.1 Overview of our results 

In this section we give an informal statement of our theorems with pointers to the relevant sections. Our 
theorem on approximating submodular functions is proved in Section 3. The definition of submodularity is 
found in the Preliminaries (Section 2). 

Informal Theorem 1.1 (Approximating submodular functions). Let a > 0,B > 0. Let f: {0, \) d — » [0, 1] be 

a submodular Junction. Then, there is an algorithm with runtime d°( lo sO/P)/ a ) w hich produces an approxi- 
mation h: {0, l} d — > [0, 1] such that Pr^m n4l/(X) - h(x)\ < a}> 1-/3. 

In Section 4 we then show how this algorithm gives the following differentially private release mecha- 
nism for Boolean conjunctions. The definition of differential privacy is given in Section 2. 

Informal Theorem 1.2 (Differentially private query release for conjunctions). Let a > 0,/5 > 0. There is an 
e- differentially private algorithm with runtime j°( 1 °g( 1 // 3 )/ Q '~) which releases the set of Boolean conjunctions 
with error at most a on a I - B fraction of the queries provided that \D\ > jO(io g (i //?)/«- _ 

The guarantee in our theorem can be refined to give an a-approximation to a 1 - B fraction of the set 
of ro-way conjunctions (conjunctions of width w) for all w e {1, d). Nevertheless, our algorithm has the 
property that the error may be larger than a on a small fraction of the queries. We note, however, that for 
8 < a p /2 our guarantee is stronger than error a in the L p -norm which is also a natural objective that has 
been considered in other works. For example, Hardt and Talwar study error bounds on mechanisms with 
respect to the Euclidean norm across all answers [HT10]. From a practical point of view, it also turns out 
that some privacy-preserving algorithms in the literature indeed only require the ability to answer random 
conjunction queries privately, e.g., [JPW09]. 

Finally, in Section 5, we study the general query release problem and relate it to the agnostic learning 
complexity in the Statistical Query model. 

Informal Theorem 1.3 (Equivalence between query release and agnostic learning). Suppose there exists an 
algorithm that learns a class C up to error a under arbitrary distributions using at most q statistical queries. 
Then, there is a release mechanism for C that makes at most 0(qd/a 2 ) statistical queries. 

Moreover, any release mechanism for C that makes at most q statistical queries implies an agnostic 
learner that makes at most 2q queries. 

2 A notable exception is the private parity-learning algorithm of [KLN + 08], which explicitly escapes the statistical query model. 
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While both reductions preserve the query complexity of the problem neither reduction preserves runtime. 
We also note that our equivalence characterization is more general than what we stated: the same proof 
shows that agnostic learning of a class C is (up to small factors) information theoretically equivalent to 
releasing the answers to all queries in a class C for any class of algorithms that may access the database 
only in some restricted manner. The ability to make only SQ queries is one restriction, and the requirement 
to be differentially private is another. Thus, we also show that on a class by class basis, the privacy cost of 
releasing the answers to a class of queries using any technique is not much larger than the privacy cost of 
simply optimizing over the same class to find the query with the highest value, and vice versa. 

Our techniques. Our release algorithm is based on a structural theorem about general submodular func- 
tions / : 2 U —> [0, 1] that may be of independent interest. Informally, we show that any submodular function 
has a "small" "approximate" representation. Specifically, we show that for any a > 0, there exist at most 
\U\ 2 / a submodular functions gi such that each satisfies a strong Lipschitz condition, and for each S c U, 
there exists an i such that f(S) = gt(S). We then take advantage of Vondrak's observation in [VonlO] that 
Lipschitz submodular functions are self -bounding, which allows us to apply recent dimension-free concen- 
tration bounds for self-bounding functions [BLM00, BLM09]. These concentration results imply that if we 
associate each function gi with its expectation, and respond to queries f(S) with E[#,-(S)] for the appropriate 
gi, then most queries are answered to within only a additive error. This yields an algorithm for learning sub- 
modular functions over product distributions, which can easily be made privacy preserving when the values 
f(S ) correspond to queries on a sensitive database. 

Our characterization of the query complexity of the release problem in the SQ model uses the multiplica- 
tive weights method [LW94, AHK05] similar to how it was used recently in [HR10]. That is we maintain 
a distribution over the universe on which the queries are defined. What is new is the observation that an 
agnostic learning algorithm for a class C can be used to find a query from C that distinguishes between the 
true data set and our distribution as much as possible. Such a query can then be used in the multiplicative 
weights update to reduce the relative entropy between the true data set and our distribution significantly. 
Since the relative entropy is nonnegative there can only be a few such steps before we find a distribution 
which provides a good approximation to the true data set on all queries in the class C. 

1.2 Related Work 

Learning Submodular Functions. The problem of learning submodular functions was recently intro- 
duced by Balcan and Harvey [BH10]; their PAC-style definition was different from previously studied 
point-wise learning approaches [GHIM09, SF08]. For product distributions, Balcan and Harvey give an 
algorithm for learning monotone, Lipschitz continuous submodular functions up to constant multiplicative 
error using only random examples. [BH10] also give strong lower bounds and matching algorithmic re- 
sults for non-product distributions. Our main algorithmic result is similar in spirit, and is inspired by their 
concentration-of-measure approach. Our model is different from theirs, which makes our results incom- 
parable. We introduce a decomposition that allows us to learn arbitrary (i.e. potentially non-Lipschitz, 
non-monotone) submodular functions to constant additive error. Moreover, our decomposition makes value 
queries to the submodular function, which are prohibited in the model studied by [BH10]. 

Information Theoretic Characterizations in Privacy. Kasiviswanathan et al. [KLN + 08] introduced the 
centralized and local models of privacy and gave information theoretic characterizations for which classes 
of functions could be learned in these models: they showed that information theoretically, the class of 
functions that can be learned in the centralized model of privacy is equivalent to the class of functions that 
can be agnostically PAC learned, and the class of functions that can be learned in the local privacy model is 
equivalent to the class of functions that can be learned in the SQ model of Kearns [Kea98]. 
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Blum, Ligett, and Roth [BLR08] considered the query release problem (the task of releasing the ap- 
proximate value of all functions in some class) and characterized exactly which classes of functions can be 
information theoretically released while preserving differential privacy in the centralized model of data pri- 
vacy. They also posed the question: which classes of functions can be released using mechanisms that have 
running time only polylogarithmic in the size of the data universe and the class of interest? In particular, 
they asked if conjunctions were such a class. 

In this paper, we give an exact information theoretic characterization of which classes of functions can 
be released in the SQ model, and hence in the local privacy model: we show that it is exactly the class of 
functions that can be agnostically learned in the SQ model. We note that the agnostic SQ learnability of 
a class C (and hence, by our result, the SQ releasability of C) can also be characterized by combinatorial 
properties of C, as done by Blum et al. [BFJ + 94] and recently Feldman [FellO]. 

Lower bounds and hardness results. There are also several conditional lower bounds on the running time 
of private mechanisms for solving the query release problem. Dwork et al. [DNR + 09] showed that under 
cryptographic assumptions, there exists a class of queries that can be privately released using the inefficient 
mechanism of [BLR08], but cannot be privately released by any mechanism that runs in time polynomial in 
the dimension of the data universe (e.g. d, when the data universe is {0, \} d ). Ullman and Vadhan [UV10] 
extended this result to the class of conjunctions: they showed that under cryptographic assumptions, no 
polynomial time mechanism that outputs a data set can answer even the set of d 2 conjunctions of two- 
literals! 

The latter lower bound applies only to the class of mechanisms that output data sets, rather than some 
other data structure encoding their answers, and only to mechanisms that answer all conjunctions of two- 
literals with small error. In fact, because there are only d 2 conjunctions of size 2 in total, the hardness result 
of [UV10] does not hold if the mechanism is allowed to output some other data structure - such a mechanism 
can simply privately query each of the d 2 questions. 

We circumvent the hardness result of [UV10] by outputting a data structure rather than a synthetic data 
set, and by releasing all conjunctions with small average error. Although there are no known computational 
lower bounds for releasing conjunctions with small average error, even for algorithms that output a data 
set, since our algorithm does not output a data set, our approach may be useful in circumventing the lower 
bounds of [UV10]. 

We also prove a new unconditional (information theoretic) lower bound on algorithms for privately 
releasing monotone conjunctions that applies to the class of algorithms that interact with the data using 
only SQ queries: no such polynomial time algorithm can release monotone conjunctions with o(l) average 
error. We note that our lower bound does not depend on the output representation of the algorithm. Because 
almost all known private algorithms can indeed be implemented using statistical queries, this provides a 
new perspective on sources of hardness for private query release. We note that information theoretic lower 
bounds on the query complexity imply lower bounds on the running time of such differentially private 
algorithms. 

There are also many lower bounds on the error that must be introduced by any private mechanism, 
independent of its running time. In particular, Kasiviswanathan et. al. [KRSU10] showed that average 
error of 0(1/ V") is necessary for private mechanisms that answer all conjunction queries of constant size. 
Recently, this work was extended by De [Del 1] to apply to mechanisms that are allowed to have arbitrarily 
large error on a constant fraction of conjunction queries of constant size. These results extend earlier results 
by Dinur and Nissim [DN03] showing that average error f2(l/ V") is necessary for random queries. 

Interactive private query release mechanisms. Recently, Roth and Roughgarden [RR10] and Hardt and 
Rothblum [HR10] gave interactive private query release mechanisms that allow a data analyst to ask a large 
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number of questions, while only expending their privacy budgets slowly. Their privacy analyses depend 
on the fact that only a small fraction of the queries asked necessitate updating the internal state of the 
algorithm. However, to answer large classes of queries, these algorithms need to make a large number of 
statistical queries to the database, even though only a small number of statistical queries result in update 
steps! Intuitively, our characterization of the query complexity of the release problem in the SQ model is 
based on two observations: first, that it would be possible to implement these interactive mechanisms using 
only a small number of statistical queries if the data analyst was able to ask only those queries that would 
result in update steps, and second, that finding queries that induce large update steps is exactly the problem 
of agnostic learning. 

2 Preliminaries 

Differential privacy and counting queries. We study the question of answering counting queries over a 
database while preserving differential privacy. Given an arbitrary domain X, we consider databases D € X". 

We write n = \D\. Two databases D — (xi x n ) and D' = (x' { , . . . , x' n ) are called adjacent if they differ only 

in one entry. That is, there exists i e [n] such that for every j + i, xj = x'.. We are interested in algorithms 
(or mechanisms) that map databases to some abstract range R while satisfying e-differential privacy: 

Definition 2.1 (Differential Privacy [DMNS06]). A mechanism M : X* —> K satisfies e-differential privacy 
if for all S CR and every pair of two adjacent databases D, D' , we have Pr(At(D) e S ) < e £ Pr(Al(D') e S). 

A counting query is specified by a predicate q: X —> [0, 1]. We will denote the answer to a counting 
query (with some abuse of notation) by q(D) = - YjxeD <lQQ . Note that a count query can differ by at 
most 1 In on any two adjacent databases. In particular, adding Laplacian noise of magnitude 1 1 en, denoted 
Lap(lfsn), guarantees e-differential privacy on a single count query (see [DMNS06] for details). 

The statistical query model and its connection to differential privacy. We will state our algorithms in 
Kearns' statistical query (SQ) model. In this model an algorithm A can access a distribution D over a 
universe X only through statistical queries to an oracle O. That is, the algorithm may ask any query q : X — > 
[0, 1] and the oracle may respond with any answer a satisfying \a - H x ~d q(x)\ < t . Here, t is a parameter 
called the tolerance of the query. 

In the context of differential privacy, the distribution D will typically be the uniform distribution over 
a data set of size n. A statistical query is then just the same as a counting query as defined earlier. Since 
SQ algorithms are tolerant to noise it is not difficult to turn them into differentially private algorithms using 
a suitable oracle. This observation is not new, and has been used previously, for example by Blum et al. 
[BDMN05] and Kasiviswanathan et al. [KLN+08]. 

Proposition 2.1. Let A denote an algorithm that requires k queries of tolerance t. Let O denote the oracle 
that outputs q(x) + Lap(k/ns). Then, the algorithm A satisfies E-differential privacy and with proba- 
bility at least 1 - fj, the oracle answers all q queries with error at most r provided that n > k ^°a k+ ^°s( l /P)) 

Proof. The first claim follows directly from the properties of the Laplacian mechanism and the composition 
property of e-differential privacy. To argue the second claim note that Pr(\Lap(cr)\ > t) < exp(-r/cr) . Using 
that cr - k/ne and the assumption on n, we get that this probability is less than f5/k. The claim now follows 
by taking a union bound over all k queries. □ 

Query release. A concept class (or query class) is a set of predicates from X — > [0, 1]. 

Definition 2.2 (Query Release). Let C be a concept class. We say that an algorithm A {a,fi)-releases C over 
a data set D if Wr q „ c {\q(D) - A(q)\ <<*}>!-/?. 
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Specifically, we are interested in algorithms which release C using few statistical queries to the under- 
lying data set. We will study the query release problem by considering the function f(q) = q{D). In this 
setting, releasing a concept class C is equivalent to approximating the function q is the following sense 

Definition 2.3. We say that an algorithm A (a, ^-approximates a function /: 2 U — > [0, 1] over a distribution 
DifPr s ~ D {\f(S)-A(S)\ <a}>\-(3. 

For many concept classes of interest, the function f(q) = q(D) will be submodular, defined next. 



Submodularity. Given a universe U, a function / : 2 U — > R is called submodular if for all S, T c U it 
holds that f(S UT) + f(S CtT) < f(S) + f(T) . We define the marginal value of x (or discrete derivative) at 
S as d x f(S) = f(SU{x})-f(S). 

Fact 2.1. A function f is submodular if and only ifd x f(S) > d x f(T)for all S QT c U and all x e U. 
Definition 2.4. A function / : 2 U -> R is y-Lipschitz if for every S c U and x € U, \d x f(S)\ < y. 



Concentration bounds for submodular functions. The next lemma was shown by Vondrak [Von 10] 
building on concentration bounds for so-called self-bounding functions due to [BLM00, BLM09]. 

Lemma 2.1 (Concentration for submodular functions). Let f: 2 U — > R be a l-Lipschitz submodular func- 
tion. Then for any product distribution D over 2 U , we have 

Pr {\f(S) -E/(5)| > t) < 2exp( \ ), (1) 

S~d xuk F \ 2(E/(S) + 5f/6)/ 

where the expectations are taken over S ~ O. 

We obtain as a simple corollary 

Corollary 2.2. Let f:2 u ^> [0, 1] be a y-Lipschitz. submodular function. Then for any product distribu- 
tion D over 2 U , we have 

s%mS) - E/( S ,| > r ,| £ 2exp(- 2([/y ^ </6) ), (2) 
where the expectations are taken over S ~ D. 



3 Approximating Submodular Functions 

Our algorithm for approximating submodular functions is based on a structural theorem, together with some 
strong concentration inequalities for submodular functions (see Lemma 2.1). The structure theorem essen- 
tially says that we can decompose any bounded submodular function into a small collection of Lipschitz 
submodular functions, one for each region of the domain. In this section, we prove our structure theorem, 
present our algorithm, and prove its correctness. 
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Algorithm 1 Decomposition for monotone submodular functions 



Input: Oracle access to a submodular function /: 2 
0. 



[0, 1] and parameter y > 



Let -< denote an arbitrary ordering of U. 
Let I <- {0} 

for x € U (in ascending order under -<) do 
I' «- 
for Bel do 

if d x f(B) > y then I' <- J' U {B U {jc}} 
J <- J U J' 

Let V(S) = {x € [/ | d x f(S) < y) denote the set of elements that have small marginal value with respect 



Output: the collection of functions Q = {g B \ B € I}, where for B e I we define the function g B : 2 V(B ^ — > 
[0,1] as g B (S)=f(S US). 



3.1 Monotone Submodular Functions 

We begin with a simpler version of the structure theorem. This version will be sufficient for approximating 
bounded monotone submodular functions from value queries, and will be the main building block in our 
stronger results, which will allow us to approximate arbitrary bounded submodular functions, even from 
"tolerant" value queries. 

Our structure theorem follows from an algorithm that decomposes a given submodular function into 
Lipschitz submodular functions. The algorithm is presented next and analyzed in Lemma 3.1. 

Lemma 3.1. Given any submodular function f:2 u ^> [0, 1] and y > 0, Algorithm 1 makes the following 
guarantee. There are maps F,T: 2 U — > 2 U such that: 

1. (Lipschitz) For every g B e Q, g B is submodular and satisfies sup^y^ sqV(B) dxQ B {S) ^ J- 

2. (Completeness) For every S c U, F(S) c 5 c V(F(S)) andg F(S \S) - f(S). 

3. (Uniqueness) For every S c U and every B € I, we have F(S) - B if and only if B c S Q V(B) and 
S n T(B) = 0. 

4. (Size) The size ofQ is at most \Q\ = \U\ mM . Moreover, given oracle access to f, we compute F, V, T 
intime\U\ 0(l/ v\ 

Note that the lemma applies to non-monotone submodular functions / as well; however, since our release 
algorithm will require the stronger condition sup xeV ^ sqv(B) \dxd(S)\ ^ % the lemma will only be sufficient 
for releasing monotone submodular functions (where it holds that \d x g(S )| < y <==> d x g(S) < y). We will 
return to the non-monotone case later. 

Proof. Algorithm 1 always terminates and we have the following bound on the size of I. 
Claim 3.2. |J| <\U\ ily 

Proof. Let B € I be a set, B - {x\, . . . , x\b\}. Let So = and Bi — {jq, . . . , x,} for i = 1, . . . , |B| - 1. Then 



to 5 c U. 



\B\-i 




(3) 



Therefore, it must be that |Z?| < l/y, and there are at most \ U\ 



/y such sets over \U\ elements. 



□ 
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Item 1 is shown next. 



Claim 3.3 (Lipschitz). For every g € Q, g is submodular and sup^mm sqV(B) d x g (S) < y. 

Proof. Submodularity follows from the fact that g B is a "shifted" version of /. Specifically, if T c S , then 
d x g B {S) = d x f(B US)< d x f{B U T) = d x g B {T), where the inequality is by submodularity of/. 

To establish the Lipschitz property, we note that by the definition of V, d x f(B) < y for every x e V(B). 
Also, by the submodularity of /, we have d x g B (S ) = d x f(B US) < d x f(B) <y. □ 

Definition of F and proof of Item 2. Now we turn to constructing the promised mappings F and T in 
order to Properties 2 and 3. Roughly, we want F(S) to choose a maximal set in I such that F(S) c S, in 
order to assure that S c V(F(S)). This task is complicated by the fact that there could be many such sets. 
We want to be able to choose a unique such set, and moreover, given any such set B, determine efficiently if 
F(S ) = B. To achieve the former task, we define a specific, deterministic mapping F(S ) and to achieve the 
latter we will carefully define the mapping T. 
We define F(S) as follows: 
let j <- 0, Bj <- 

for x € U (in ascending order under -<) do 

if x £ V(Bj) and x e S then B M <- Bj U {x\, j <- j + 1 
return F(S) = Bj. 

Note that this procedure is similar to the procedure we use to construct I. To construct J, we gradually 
constructed a tree of sets, where each set B € I had a child for every set BUjij such that x has high influence 
on B (x £ V(B).. The procedure F(S) differs in that it only constructs a single root-leaf path in this tree, 
where for each Bj in the path, the next set in the path is Bj U {x} where x is the minimal x e S that has high 
influence on Bj (and has not already been considered by F(S). We will use P(S) = (Bq c B\ c • • • c F(S)) 
to denote this path, which is the sequence of intermediate sets Bj in the execution of F(S). Given these 
observations, we can state the following useful facts about F. 

Fact 3.1. IfF{S) = B, then P(S) - P{B). Moreover, for every S eU,P{S)Q J. 

We can now establish Property 2 by the following claim. 

Claim 3.4 (Completeness). For every S QU, F(S)QS c V(F(S)), and g F(S) (S) - f(S). 

Proof. Let P(S) = Bq c B\ c • • • c F(S). F(S) always checks that x e S before including an element x, 
so F(S) c S . To see that S c V(F(S)), assume there exists x e S \ V(F(S)). By submodularity we have 
d x f(Bj) > d x f(F(S)) > y for every set Bj. But if d x f(Bj) > y for every Bj and x e S , it must be that 
x e F(S). But then d x f(F(S)) = 0, contradicting the fact that x $ V(F(S)). 

Finally, we note that since S c V(F(S)), g F(S) (S) is defined (S is in the domain of g F(S ) ) and since 
F(S)cS,g FiS \S) = f(F(S)US) = f(S). □ 

Definition of T and proof of Item 3. We will now define the mapping T. The idea is to consider a set 
Bel and P(B) and consider all the elements we had to "reject" on the way from the root to B. We say that 
an element x e U is "rejected" if, when x is considered by F(S), it has high influence on the current set, but 
is not in B. Since any set S such that B = F(S) satisfies P(S) = P(B) (Fact 3.1), and any set 5 that contains 
a rejected element would have taken a different path, we will get that the elements x e T(B) "witness" the 
fact that B + F(S). We define the map T(B) as follows: 

let j ^0,Bj^Q,R^Q 

for x € U (in ascending order under -<) do 
if x <£ V(Bj) mdx<£B then R <- R U {x) 
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else if x & Vg and x e B then Bj+i <— Bj U {x}, j <— j + 1 
return 7/(fi) = 7?. 
We'll establish Property 3 via the following two claims. 

Claim 3.5. IfB = F(S), then B c S c V(fl) and 5 n 7/(5) - 0. 

Proof. We have already demonstrated the first part of the claim in Claim 3.4, so we focus on the claim that 
5 n T(B) = 0. By Fact 3.1, every set S s.t. B = F(S) satisfies P(S) = P(B). Let (B c fii c • • • c B) = P(B). 
Suppose there is an element x e S n 7/(5). Then there is a set By such that x g V(Bj) and * £ 5. But since 
x £ ^(5^) and x e S , it must be that x e Bj +i , contradicting the fact that Bj + \ c 5. □ 

Now we establish the converse. 
Claim 3.6. IfB c 5 c V(B), S n 7/(5) = 0, 5 - F(S). 

Proof. Suppose for the sake of contradiction that there a set B' + B such that B' - F(S). There exists an 
element x e BaB', and we consider the minimal such x under < Let P(B) = (Bq c B\ c • • • c B) and 
P(S) = P{B') - (B' Q cfiJC"'C S'). Since x is minimal in BaB', there must be ; be such that B { = B\ for 
all i < j, but x e Bj+iAB'j +v Consider two cases: 

1. B d B' . Thus x e B \ B' Moreover, since x € B c S, it must be that when x was considered in 

the execution of F(S), and B' f was the current set, it was the case that x e V(B'-). But Bj = B'-, so 

j j J j 

x e V(Bj), contradicting the fact that x e Bj + \. 

2. B t> B' . Thus x e B' \ B. Since x e B' = F(S) Q S (Claim 3.4), we have x e 5. Moreover, since 
x e B'j +1 we must have x g V(B^) = Thus we have x ^ V(Bj) and x <£ B, which implies 
x e by construction. Thus S n T(B) ^ 0, a contradiction. 

□ 

The previous two claims establish Item 3. 

Finally we observe that the enumeration of I requires time at most \U\ • \I\ = \U\°^ l ^\ since we iterate 
over each element of U and then iterate over each set currently in I. We also note that we can compute 
the mappings F and T in time linear in \I\ = \U\°^l y ^ and can compute V(B) in time linear in \U\. These 
observations establish Property 4 and complete the proof of Lemma 3.1. □ 

Lemma 3.7 (Lemma 3.1 with tolerance). Given any submodular function /: 2^ — > [0, 1] and y > 0, 
Algorithm 2 makes the following guarantee. There are maps F, T: 2 U —> 2 U satisfying properties 1-4 of 
Lemma 3.1 and moreover, can be computed using tolerant queries to f with tolerance 7/ 12. 

Proof. Throughout the proof, we will assume that the oracle always gives the same answer to each query. 
Thus the function / defined in Algorithm 2 is well defined. Note that f(S ) need not be submodular even if / 
is, however, we can assume that we have exact oracle access to f(S). Also note that, since we can compute 
d x f(S) using two queries to /, we are guaranteed that for every S c U, and x e U, 

\dJ(S)-d x f(S)\< 7 /6. (4) 

Observe that Algorithm 2 differs from Algorithm 1 only in the choice of parameters. The analysis 
required to establish the Lemma is also a natural modification of the analysis of Lemma 3.1, so we will refer 
the reader to the proof of that Lemma for several details and only call attention to the steps of the proof that 
require modification. 
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Algorithm 2 Decomposition for monotone submodular functions from tolerant queries 

Input: Tolerant oracle access to a submodular function /: 2 U — > [0, 1] with tolerance at most 7/ 12 and 
parameter y > 0. 

Let / denote the function specified by tolerant oracle queries to / such that for every S c U 

1/(5)- f(S)\ <yl 12. 



Let -< denote an arbitrary ordering of U. 
Let I <— {0} 

for x € U (in ascending order under -<) do 
I' <- 
for B e I do 

if dJ{B) > y/3 then J' <- J' U {B U {x}} 
J<-IuJ' 

Let V(5) = {x e U \ d x f(S) < 2y/3} denote the set of elements that have small marginal value with 
respect to S c U. 

Output: the collection of functions Q = {g B \ B € I}, where for B e I we define the function g B : 2 V(B ^ — > 
[0,1] asg B (S)=f(S US). 



We will proceed by running through the construction of Lemma 3.1 on f(S) using y/3 as the error 
parameter. Since the argument is a fairly straightforward modification to Lemma 3.1, we will refer the 
reader to the proof of that Lemma for several details, and only call attention to the steps of the proof that 
require modification. 

First, we establish a bound on the size of I 

Claim 3.8. |J| < \U\ 6/ ? 

Proof. Let B € I be a set, B = {x\, . . . , x\b\}- Let Bo = and B; = {x\, . . . , x,} for i = 1, . . . , |B| - 1. Then 
|£|-l 1*1-1 

1 > f(B) = ^ d XM f(Bd > J] (d XM m) - y/6) > \B\ ■ (7/3 - 7/6) - \B\ ■ 7/6 . (5) 

i=Q ;=0 

Therefore, it must be that |B| < 6/7, and there are at most \ U\ 6 ^ 7 such sets over \U\ elements. □ 
Item 1 is shown next. 

Claim 3.9 (Lipschitz). For every g B € Q, g B is submodular and ^PxeV{B)^cv{B) d x g B {S) < 7 

Proof. The proof of submodularity is identical to Claim 3.3 

To establish the Lipschitz property, observe that for every B c U, and every x e V(B), d x f(B) < 
dJ(B) + y/6<y. □ 

Definition of F and proof of Item 2. In addition to the sets V(B) = {x e U \ d x f(B) < 2y/3), we will 
define the sets V'(B) = {x e U \ dJ{B) < y/3), note that for every B Q U, V'(B) c V(B). We define the 
promised mapping F(S) in the the same manner as in the proof of Lemma 3.1, but we use V in place of V 
to decide whether or not we select an element x for inclusion in the set F(S). 

Now we establish Property 2 via the following claim, analogous to Claim 3.4 in the proof of Lemma 3.1 
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Claim 3.10 (Completeness). For every S Q U, F(S)QS c V(F(S)). Moreover, g F{S) (S) = f(S). 



Proof. Let P(S) = B c B x c • • • c F(5). The fact that F(S) Q S follows as in Claim 3.4. To see that 
S c V(F(S)), assume there exists x e S \ V(F(S)). By submodularity of /, and (4), we have 

dJ(Bj) > d x f(Bj) - y/6 > d x f(F(S)) - y/6 > dJ(F(S)) - y/3 > y/3. 

Thus, dJ{Bj) > y/3 for every set By. But if d x f(Bj) > y/3 for every By and x e 5 , then x $ V'(Bj) for 
every By, and it must be that x e F(S). But then d x f(F(S)) = 0, contradicting the fact that x £ V(F(S)). 
The fact that g F(S) (S ) = f(S) follows as in the proof of Claim 3.4. □ 



Definition of T and proof of Item 3. We also define the promised mapping T(S ) in the same manner as 
in the proof of Lemma 3.1, but using V in place of V to decide whether or not we select an element x for 
inclusion in the set F(S). 

To establish Property 3, we note that the proofs of Claims 3.5 and 3.6 do not rely on the submodular- 
ity of /, therefore they apply as-is to the case where we compute on /, even though / is not necessarily 
submodular. 

Property 4 also follows as in the proof of Lemma 3.1. This completes the proof of the Lemma. □ 

We now present our algorithm for learning monotone submodular functions over product distributions. 
For a subset of the universe V c U, let Dy denote the distribution D restricted to the variables in V. Note 
that if D is a product distribution, then Dy remains a product distribution and is easy to sample from. 

Algorithm 3 Approximating a monotone submodular function from tolerant queries 
Learn(/,tt,yS, D) 

^7 - 6 log(2//J) • 

Construct the collection of functions Q returned by Algorithm 2 and let F, V, T be the associated map- 
pings given by Lemma 3.7 with parameter y. 
Estimate the value n gB = B s ^!o vmnB) [g B (S)] for each g B e Q. 

Output the data structure h that consists of the values n g s for every g B e Q as well as the mapping F . 



Theorem 3.11. For any a,B e (0, 1], Algorithm 5 (a,/3)-approximates any submodular function f:2 u ^> 
[0, 1] under any product distribution D in time \U\°^- a '"s^ 1 // 3 )) using oracle queries to f of tolerance 
a 2 /721og(2/yS). 

Proof. For a set S c JJ, we let B = F(S ) and g B be the corresponding submodular function as in Lemma 3.7. 
Note that since the queries have tolerance a 2 /721og(l/6) < y/12, the lemma applies. We will analyze the 
error probability as if the estimates p g B were computed using exact oracle queries to /, and will note that 
using tolerant queries to / can only introduce an additional error of a 2 /721og(l//?) < a /6. We claim that, 
under this condition 

Pr {\f{S) - h(S)\ > 5a/6] = Pr (|/ (S) (5) -/V<«l > 5^/6} 

To see this, recall that for every S c U, g F(S \S) = f(S). By Property 3 of Lemma 3.1, the condition that 
B = F(S) is equivalent to the conditions that B c 5 c V(B) and S n T(B) = 0. Hence, 

Pr J\g B (S)-p g s\ > 5a 1 6 I B = F{S)\ - Pr \\g B {S)-p gB \ > 5a/ 6} . 
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Now, applying the concentration inequality for submodular functions stated as Corollary 2.2, we get 



Plugging in t = 5a/6y = 5log V /B) and simplifying we get Pts~d 7(B)V(B) ||# B (S) - p gB \ > a} < B . Combining 
this with (6), the claim follows. □ 



3.2 Non-monotone Submodular Functions 

For non-monotone functions, we need a more refined argument. Our main structure theorem replaces Prop- 
erty 1 in Lemma 3.1 by the stronger guarantee that \d x g(S)\ < a for all g e Q, even for non-monotone 
submodular functions. Observe that for a submodular function / : 2 V — > R, the function f:2 v — > R 
denned as f(S) = f(V\S) is also submodular; moreover 

M dJ(S) = - sup d x f(S). (8) 

xeV,SQV xeV.SCV 

Given these two facts, we can now prove our main structure theorem. 



Algorithm 4 Decomposition for submodular functions from tolerant queries 

Input: Tolerant oracle access to a submodular function /: 2^ — > [0, 1] with tolerance at most y/12 and 
parameter y > 0. 

Let / denote the function specified by tolerant oracle queries to / such that for every S c JJ 

1/(5)- f(S)\ <yl 12. 



Let -< denote an arbitrary ordering of U. 

Let 0(f) denote the collection of functions returned by Algorithm 2 with oracle / and parameter y, and 
let Ff, Vf, Tf be the associated mappings promised by Lemma 3.7. 
for g B e §(f) do 

Let Q(B) be the collection of functions returned by Algorithm 2 with oracle g and parameter y, and 

let Fb, Vb, Tb be the associated mappings promised by Lemma 3.7. 
Let V(S, T) = Vf(S) n Vs(T) denote the set of elements that have small marginal absolute value with 
respect to S, T c JJ. 

Output: the collection of functions § = Ug B e@(f){g B ' C = g C \<f e Q(B)\ where g B - c : 2 y(BC) [0, 1], 



Theorem 3.12. Given any submodular function / : 2 — > [0, 1] an<i y > 0, Algorithm 4 makes the following 
guarantee. There are maps F : 2 U — > 2^ x2 y T: 2 U x2 u ^> 2 U such that: 

1. (Lipschitz) For every B ' C e ^, g B C is submodular and satisfies sup te y (B>C) ,sqV(b,q l^xd 8 (S)\ < y. 

2. (Completeness) For every S c U, F(S) c 5 c V(F(5)) andg F(S \S) = f(S). 

3. (Uniqueness) For every o s - c e ^, F(S) = (B, C) if and only ifB, CQS c V(fl, C) andSnT{B, C) = 0. 

4. (Size) 77ie size o/^ is at most \Q\ = \U\ 0{l/y) . Moreover, given tolerant oracle access to f with 
tolerance y/12, we compute F, V, T in time \ U\°^ l ^ y \ 

Proof. First we show Item 1 
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Claim 3.13 (Lipschitz). For every g B - c e Q, g B ' C is submodular and sup rel/(g c -, sqv(b,c\ \dx9 B ' C (S )l ^ J- 

Proof. Submodularity follows directly from Property 1 of Lemma 3.7. The same property of the lemma 
guarantees that for every g B e §{f) and g c e §{B), sup reVs(C)Sc y B(C) d x g c (S) < y. Moreover, by (8), 
inf. xev f (B).SQV f {B) d x g B (S) > -y. Taken together, we obtain sup jreV(B C) 5 qv(B.C) \d x g B,C (S)\ <y. □ 

Definition of F and proof of Item 2. Item 2 will follow from the analogous property in Lemma 3.7 almost 
directly. To construct the mapping F(S), we want to first compute the appropriate function g B e 0(f), 
using Ff(S) and then find the appropriate function g c e Q{B) using Fb(S). Thus we can take F(S) = 
(Ff(S),Fp f (s)(S))- By Lemma 3.7, Item 2 we have B c S c Vf(B) and C c S c Vb(C), so we conclude 
B,CQS c C). 

Definition of T and proof of Item 3. Item ?? will also follow from the analogous property in Lemma 3.7. 
By Lemma 3.7, Item 3, we have that F f (S) = B if and only if B c 5 c V/(S) and S n 7/(5) = 0. By the 
same Lemma, we also have that F B (S ) = C if and only if C c 5 c Vg(C) and 5 n Tb(C) = 0. So if we 
define T(B, C) = T f (B) U T B (C), we can conclude that F(S) = (B, C) if and only if B, C c 5 c V(fl, C) and 
5 n r(S,C) = 0. 

Now it is clear that F(S) = (B,C) if F f (S) = B and F B (S) = C, which by Property 3 of Lemma 3.7 
necessitates that B c S c V/(B), S n 7/(5) = 0, C c 5 c V B (5), and 5 n T B (C) = 0. We have already 
defined V(B, C) and now we define T(B, C) = T f (B) U T B (C). It is clear now that F(S) = (B, C) if and only 
if B, C c S c V(fl, C) and 5 n T(B, C) - 0. 

The size of Q and running time bounds in Property ?? also follow directly from the analogous property 
of Lemma 3.1. The fact that we can compute the family Q and the associated mappings F, V, T using oracle 
access to / with tolerance 7/ 12 follows from the fact that each invocation of Lemma 3.1 can be computed 
using queries with tolerance 7/ 12 and from the fact that Algorithm 4 only queries / in order to invoke 
Lemma 3.7. This completes the proof of the Theorem. □ 

We now present our algorithm for learning arbitrary submodular functions over product distributions. 
For a subset of the universe V Q C, let Dy denote the distribution D restricted to the variables in V. Note 
that if D is a product distribution, then Dy remains a product distribution and is easy to sample from. To 

Algorithm 5 Approximating a non-monotone submodular function 
LearnC/>,j6, D) 

y et -v - ^ 

^7 - 61og(2//?)- 

Construct the collection of functions Q and the associated mappings F, V, T given by Theorem 3.12 with 
parameter y. 

Estimate the value ^l cj b,c = Es^ V(BC ^ T(BC) [g B ' C (S)] for each g BC e Q. 

Output the data structure h that consists of the values /u g B.c for every g B ' C e Q as well as the mapping F. 



avoid notational clutter, throughout this section we will not consider the details of how we construct our 
estimate fi g . However, it is an easy observation that this quantity can be estimated to a sufficiently high 
degree of accuracy using a small number of random samples. 

Theorem 3.14. For any a,B e (0, 1], Algorithm 5 (a,B)-approximates any submodular function f:2 u ^> 
[0, 1] under any product distribution in time \ U\ 0( - a "® '™ using oracle queries to f of tolerance a 1 112 log(l/yS) 

Proof. For a set S c U, we let (B,C) = F(S) and g BC be the corresponding submodular function as in 
Theorem 3.12. Note that since the queries have tolerance a 2 /721og(l/yS) < y/12, the lemma applies. We 
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will analyze the error probability as if the estimates p g B were computed using exact oracle queries to /, and 
will note that using tolerant queries to / can only introduce an additional error of a 2 /721og(l//?) < a/6. We 
claim that, under this condition We claim that 



Pr {\f(S) - h(S)\ > 5a /6} = Pr f|/ (S) (5) - u^A > 5a/6\ 



= s p l I(B ' C) = F(S)] ■ s Pr v {\ gB ' C(S) -^M > 5a/6 1 (S ' C) = F{S) ) ■ 

(9) 

To see this, recall that for every S c U, g Ft - s \S) = f(S). By Property 3 of Lemma 3.1, the condition that 
B = F(S ) is equivalent to the conditions that B, C c S c V(B, C) and S n 7\fi, C) = 0. Hence, 

Pr [\g^ c (S)-fi g s,c\ > 5a/6 \ (B,C) = F(S)} = Pr l\g B (S) -pt B \ > 5a/6\ . 

S~D S ~T>V(B,0\T{B,C) 

Now, applying the concentration inequality for submodular functions stated as Corollary 2.2, we get 



Pr (|/' c (5) - /v,c| > yt) < 2expl-— — — — I . (10) 

s~v V(b ,c)\t(b,c) [ ' \ 2(l/y + 5t/6)) 

Plugging in t = 5a/6y = ^ and simplifying we get Pr s ^ Dv(BCmBQ {\g B - c (S) - n gB .c\ > a] < B. 

Combining this with Equation (9), the claim follows. □ 



t 
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4 Applications to privacy-preserving query release 

In this section, we show how to apply our algorithm from Section 3 to the problem of releasing monotone 
conjunctions over a boolean database. In Section 4.1, we also show how our mechanism can be applied to 
release the cut function of an arbitrary graph. 

Let us now begin with the monotone disjunctions. We will then extend the result to monotone conjunc- 
tions. Given our previous results, we only need to argue that monotone disjunctions can be described by 
a submodular function. Indeed, every element S e {0, \ } d naturally corresponds to a monotone Boolean 
disjunction ds '■ {0, 1}'' — > {0, 1} by putting 

ds(x) d = \J x t . 

i: 5(i)=l 

Note that in contrast to Section 3 here we use x to denote an element of {0, \ } d . Let Foi s j : {0, \} d — * [0, 1] be 
the function such that / 7 Disj(^) = ds(D). It is easy to show that Fd^(S) is a monotone submodular function. 

Lemma 4.1. Foisi i s a monotone submodular function. 

Proof. Let denote the set of elements x e D such that Xj = 1, and let XT denote the set of elements 
x e D such that x,- - 0. Consider the set system U = {Xt ,XT) c ! =l over the universe of elements x e D. Then 
there is a natural bijection between Foisj(O) and the set coverage function Cov : 2 U -> [0, \D\] defined to be 
Cov(5) = | {JxeuX\> which is a monotone submodular function. □ 

We therefore obtain the following corollary directly by combining Theorem 3.11 with Proposition 2.1. 

Corollary 4.2. Let a,B,s > 0. There is an s-differentially private algorithm that (a,B)-releases the set of 
monotone Boolean disjunctions over any product distribution in time d t( - a ' B) for any data set of size \D\ > 
d ,{a < 3) Is where t{a,B) - 0{a' 2 log(l//?)). 
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Algorithm 6 Privately Releasing Monotone Disjunctions 
Release(D,a,/3, e, £>) 

Simulate the oracle queries Foi S j(S) by answering with ds(D) + Lap(t(a,fi)/s\D\). 

^7- 61og(2/y3)- 

Construct the collection of functions Q and the associated mappings F, V, T given by Lemma 3.7 on the 
function Fuisj with parameter y. 

Estimate the value n g B = Es~D vmT(B) [g B (S)] for each g B e Q. 

Output the data structure h that consists of the estimated values p, g B for every g B e Q as well as the 
mapping F. To evaluate any monotone disjunction query ds(D), compute fJ. g F{s). 



For completeness, we will present the algorithm for privately releasing monotone disjunctions over a 
product distribution D for a data set D, though we will rely on Corollary 4.2 for the formal analysis. 

We will next see that this corollary directly transfers to monotone conjunctions. A monotone Boolean 
conjunction cs '■ {0, 1 } d — > {0, 1 } is defined as 

C 5 «= f /\x,= 1- \/(l-X ; ). 

ieS ieS 

Given the last equation, it is clear that in order to release conjunctions over some distribution, it is sufficient 
to release disjunctions over the same distribution after replacing every data item x e D by its negation x, i.e., 
Xi - 1 - Hence, Corollary 4.2 extends directly to monotone conjunctions. 

Extension to width w. Note that the uniform distribution on disjunctions of width w is not a product 
distribution, which is what we require to apply Theorem 3.14 directly. However, in Lemma 4.3 we show 
that for monotone submodular functions (such as ^D is j) the concentration of measure property required in 
the proof Theorem 3.14 is still satisfied. Of course, we can instantiate the theorem for every we { 1, . . . , k] 
to obtain a statement for conjunctions of any width. 

Indeed, given a monotone submodular function /: 2 U —> R, let S e 2 U be the random variable where 
for every x € U, independently x e S with probability w/d and x <f. S with probability 1 - w/d. On the other 
hand, let T e2 u denote the uniform distribution over strings in 2 U of weight w. The following lemma is due 
to Balcan and Harvey [BH10]. 

Lemma 4.3. Assume f : 2 U — > R is monotone function, and S and T are chosen at random as above. Then, 

Pr[/(r)>T]<2Pr[/(5)>r] (11) 

Pr[/(D<r] <2Pr[/(5)<r] (12) 

Remark 4.1. Throughout this section we focus on the case of monotone disjunctions and conjunctions. 
Our algorithm can be extended to non-monotone conjunctions/disjunctions as well. However, this turns out 
to be less interesting than the monotone case. Indeed, a random non-monotone conjunction of width w is 
false on any fixed data item with probability 2~ w , thus when w > log(l/a), the constant function is a 
good approximation to Fpisj on a random non-monotone conjunction of width w. We therefore omit the 
non-monotone case from our presentation. 

4.1 Releasing the cut function of a graph 

Consider a graph G = (V, E) in which the edge-set represents the private database (We assume here that each 
individual is associated with a single edge in G. The following discussion generalizes to the case in which 
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individuals may be associated with multiple edges, with a corresponding increase in sensitivity). The cut 
function associated with G is Jq : 2 V — > [0, 1], defined as: 

f G (S) = -^-\{(u,v)eE:ueS,vtS}\ 
|VT 

We observe that the graph cut function encodes a collection of counting queries over the database E and so 
has sensitivity 1/|V| 2 . 

Fact 4.1. For any graph G, fc is submodular. 

Lemma 4.4. The decomposition from Theorem 3.12 constructs a collection offiinctions Q of size \Q\ < 2 2 ^ a . 

Proof. Let u e V, and S c V such that |<9 h /g(^)I ^ os. It must be that the degree of u in G is at least 
a ■ \E\. But there can be at most 2/a such high-influence vertices, and therefore at most 2 2 ^ a subsets of high 
influence vertices. □ 

Corollary 4.5. Algorithm 5 can be used to privately (a,fi)-release the cut function on any graph over any 
product distribution in time t(a,fi,s)for any database of size \D\ > t(a,/3,s), while preserving s-differential 
privacy, where: 

2 0(a- 2 logCl/^S)) 

t(a,p,s) = 

E 

Proof. This follows directly from a simple modification of Theorem 3.14, by applying Lemma 4.4 and 
plugging in the size of the decomposition Q. The algorithm can then be made privacy preserving by applying 
proposition 2.1. □ 



5 Equivalence between agnostic learning and query release 

In this section we show an information-theoretic equivalence between agnostic learning and query release 
in the statistical queries model. In particular, given an agnostic learning algorithm for a specific concept 
class we construct a query release algorithm for the same concept class. 

Consider a distribution A over X x {0, 1} and a concept class C. An agnostic learning algorithm (in 
the strong sense) finds the concept q e C that approximately maximizes ^(x^-a {q(x) - b) to within an 
additive error of a. Our reduction from query release to agnostic learning actually holds even for weak 
agnostic learning. A weak agnostic learner is not required to maximize Pr (r ^y.^ {<?(*) = b}, but only to find 
a sufficiently good predicate q provided that one exists. 

We use STAT r (A) to denote the statistical query oracle for distribution A that takes as input a predicate 
q : X — > {0, 1} and returns a value v such that \v - E v ^[^r(x)]| 

Definition 5.1 (Weak Agnostic SQ-Learning). Let C be a concept class and y, t > and < /? < a < 1/2. 
An algorithm 3\ with oracle access to STAT r (A) is an (a,yS, y, x)-weak agnostic learner for C if for every 
distribution A such that there exists q* e C satisfying Pr^^A {q*{x) = b) > ^/i+a , Jl(A) outputs a predicate 
q : X —> {0, 1! such that Pr^b)~A {<?(*) = b] > l /2 +/?, with probability at least 1 - y. 

Note that if we can agnostically learn C in the strong sense from queries of tolerance r to within additive 
error a - f3 with probability 1 - y, then there is also an {a,f3, y, r)-weak agnostic learner. 

We are now ready to state the main result of this section, which shows that a weak agnostic SQ- learner 
for any concept class is sufficient to release the same concept class in the SQ model. 

Theorem 5.1. Let C be a concept class. Let 3\ be an algorithm that (a/2,/3, y, r) weak agnostic-SQ learns 
C with t < yS/8. Then there exists an algorithm S that invokes 3i at most T — 81og|X|/yS 2 times and 
(a, 0)-releases C with probability at least 1 - Ty. 
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Algorithm 7 Multiplicative weights update 



Let D denote the uniform distribution over X. 
For? - l,...,T = r81og|X|/^ 2 l + 1: 



Consider the distributions 



A? = i/2(D,l) + i/2(D f _i,0) 



a: = 



1/2(D,0) + 1/2(A-1,D. 



Let q+ = ${{Af) and = &l(Aj). Let be the value returned by STAT T (A+) on the query qf and be 
the value returned by STAT T (A~) on the query q~. Let v t = max^, vj} - 1/2 and q t be the corresponding 
query. 
If: 



proceed to "output" step. 

Update: Let D t be the distribution obtained from D t -\ using a multiplicative weights update step with 
penalty function induced by q t and penalty parameter 77 = yS/2 as follows: 



Output a c = ~E x ~d t c{x) for each c e C. 

The proof strategy is as follows. We will start from Do being the uniform distribution over X. We will 
then construct a short sequence of distributions D\,Dj, . . . ,Dj such that no concept in C can distinguish 
between D and Dj up to bias a. Each distribution D, is obtained from the previous one using a multiplicative 
weights approach as in [HR10] and with the help of the learning algorithm that's given in the assumption 
of the theorem. Intuitively, at every step we use the agnostic learner to give us the predicate q, e C which 
distinguishes between D t and D. In order to accomplish this we feed the agnostic learner with the distribution 
A t that labels elements sampled from D by 1 and elements sampled from D t by 0. For a technical reason we 
also need to consider the distribution with and 1 flipped. Once we obtained q t we can use it as a penalty 
function in the update rule of the multiplicative weights method. This has the effect of bringing D and D t 
closer in relative entropy. A typical potential argument then bounds the number of update steps that can 
occur before we reach a distribution D t for which no good distinguisher in C exists. 

5.1 Proof of Theorem 5.1 

Proof. We start by relating the probability that q t predicts b from x on the distribution A r + to the difference 
in expectation of q t on D and D t ~\. 

Lemma 5.2. For any q: X — > {0, 1}, 




(13) 



D' t {x) = e\p(qq,(x)) ■ D t -\(x) 



D t (x) = 



ZxexD'tix) 




(14) 
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Proof. If q t = qf then 



Pr {q(x) =b) = \ mq{x) = 1} + - Pr {q(x) = 0} 

(x,b)~A+ 2 x~D I x~D,- l 

= \ E + - E [l-q{x)\ 

Z x~D z x~D,-[ 



- - + - I E g(x) 



E g(x) 

Note that Pr (jr ^ Ar {g(x) = b] = 1 - Pr (jc , fc) .. A; - {<?(x) = (1 - &)} = 1 - Pr (x ^ A +{g(x) - b), so if ? f - q~ then 



Pr W(x) - b) = 1 - Pr {q(x) = b] = 1 - I - - - I E <j(x) 



2 2 U~d 



E ^r(x) 

x~D t -i 



J + E - E q(x) 

L \x~D x~D,-i 



The rest of the proof closely follows [HR10]. For two distributions P, Q on a universe X we define the 
relative entropy to be RE(P||0 = zZxex P( X )\°2,(P( X )/ Q( x ))- We consider the potential 

>F f = RE(D||A) . 



Fact 5.1. % > 
Fact 5.2. T < log \X\ 

We will argue that in every step the potential drops by at least yS 2 /4 Hence, we know that there can be at 
most 4 log \X\/a 2 steps before we reach a distribution that satisfies (13). 

The next lemma gives a lower bound on the potential drop in terms of the concept, q t , returned by the 
learning algorithm at time t. Recall, that q (used below) is the penalty parameter used in the multiplicative 
weights update rule. 



Lemma 5.3 ([HR10]). 



Let 



opt f = sup 



E q t (x) - E q t (x) 

x~D x~D,-i 



Pr {q{x)=b)-- 

(x,b)~A+ 2 



(15) 



Note that Pr(^)^ A -{?(x) = b} - 1 - Pt( x j>)~a+{~~ ,< 1(. x ) = b). For the remainder of the proof we treat the two 
cases symmetrically and only look at how far from 1 /2 these probabilities are. The next lemma shows that 
either opt r is large or else we are done in the sense that D t is indistinguishable from D for any concept from 
C. 



Lemma 5.4. Let a > 0. Suppose 
Then, for all q e C, 



a 

opt t < ^ 



E q(x) - E q t (x) 

x~D x~D, 



< a 



(16) 
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Proof. From Lemma 5.2 we have that for every q € C 



£ > opt, > Pr \q(x) = b} - i = - I E <?(x) - E 



Thus a > (E^d #(x) - E^d ( <?/(x)). Similarly, 



£ > opt, > Pr {q{x) = b] - I = \ I E <?(x) - E q t {x) 

2 (x,b)~Ay 2 2 \x~D t x~D 



Thus -a < (E A ~z) q{x) - ~E x ~d, qt(x))- So we conclude a > |e v ~,d q(x) - H x ~d, <?r(x)| 



□ 



We can now finish the proof of Theorem 5.1. By our assumption, we have that so long as opt, > a/2 the 
algorithm 3\ produces a concept q t such that with probability 1 - y 



Pr [q t (x) = b]-- 

{x,b)~A+ 2 



(17) 



For the remainder of the proof we assume that our algorithm returns a concept satisfying Equation (17) in 
every stage for which opt, > a/2. By a union bound over the stages of the algorithm, this event occurs with 
probability at least 1 - Ty. 

Assuming Equation (13) is not satisfied we have that 



^<^-2t<u,-t< 
4 2 



Pr{^r f (x) = b) 

A* 



The leftmost inequality follows because r < /3/8. We then get 



>rj 



Eg,(x) - E g,(x) 

D D,-i 

4Pr{<7,(x) = b}-2 

A, 



-v 



- V 



/3 2 /3 2 



> 



2 4 

£ 

4 



(Lemma 5.3) 

(Lemma 5.2) 
(Equation 13 not satisfied) 
07=j8/2) 



Hence, if we put T > 4 log \X\I0 1 , we must reach a distribution that satisfies (13). But at that point, call 
it t, the subroutine 3[ outputs a concept q t such that 



Pr (g t (x) = b)-- 

(x,b)~A+ 2 



<V, + T<^- + T<B 

2 



In this case, by our assumption that Equation 17 is satisfied whenever opt, > 1/2 + a/2, we conclude that 
opt, < 1/2 + a/2. By Lemma 5.4, we get 



sup 

qeC 



E q(x) - E q t (x) 

x~D x~D, 



< a. 



But this is what we wanted to show, since it means that our output on all concepts in C will be accurate up 
to error a. □ 
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We remark that for clarity, we let the failure probability of the release algorithm grow linearly in the 
number of calls we made to the learning algorithm (by the union bound). However, this is not necessary: 
we could have driven down the probability of error in each stage by independent repetition of the agnostic 
learner. 

This equivalence between release and agnostic learning also can easily be seen to hold in the reverse 
direction as well. 

Theorem 5.5. Let C be a concept class. If there exists an algorithm S that (a, 0)-releases C with probability 
1 - y and accesses the database using at most k oracle accesses to STAT T (A), then there is an algorithm 
that makes 21c queries to STAT r (A) and agnostically learns C in the strong sense with accuracy 2a with 
probability at least 1 - 2y. 

Proof. Let Y denote the set of examples with label 1 , and let N denote the set of examples with label 0. We 
use STAT T (A) to simulate oracles STAT T (F) and STAT T (A f ) that condition the queried concept on the label. 
That is, STAT r (F), when invoked on concept q, returns an approximation to Pr^l^W = I A (x € Y)} and 
STAT r (A r ) returns an approximation to Pr^A {<?(■*) = I A (x € Y)]. We can simulate a query to either oracle 
using only one query to STAT T (A). 

Run S(Y) to obtain answers a\, . . . , a^, and run S(N) to obtain answers a^, . . ., a^,,. Note that this 
takes at most 2k oracle queries, using the simulation described above, by our assumption on S. By the union 
bound, except with probability 2y, we have for all q\ e C: \cjj(Y) - aj\ < a and \qi(B) - af \ < a. Let 
q* - arg max^gcfaf - af). Observe that q*(D) > max^c q(D) - 2a, and so we have agnostically learned C 
up to error 2a. □ 

Feldman proves that even monotone conjunctions cannot be agnostically learned to subconstant error 
with polynomially many SQ queries: 

Theorem 5.6 ([FellO]). Let C be the class of monotone conjunctions. Let k(d) be any polynomial in d, the 
dimension of the data space. There is no algorithm which agnostically learns C to error o(l) using k(D) 
queries to STATi/^. 

Corollary 5.7. For any polynomial in d, k(d), no algorithm that makes k(d) statistical queries to a database 
of size k(d) can release the class of monotone conjunctions to error o{\). 

Note that formally, Corollary 5.7 only precludes algorithms which release the approximately correct 
answers to every monotone conjunction, whereas our algorithm is allowed to make arbitrary errors on a 
small fraction of conjunctions. 

Remark 5.1. It can be shown that the lower bound from Corollary 5. 7 in fact does not hold when the accu- 
racy requirement is relaxed so that the algorithm may err arbitrarily on 1% of all the conjunctions. Indeed, 
there is an inefficient algorithm (runtime poly(2 rf )J that makes poly(<i) statistical queries and releases ran- 
dom conjunctions up to a small additive error. The algorithm roughly proceeds by running multiplicative 
weights privately (as in [HR10] or above) while sampling, say, 1000 random conjunctions at every step 
and checking if any of them have large error. If so, an update occurs. We omit the formal description and 
analysis of the algorithm. 

We also remark that the proofs of Theorems 5.1 and 5.5 are not particular to the statistical queries model: 
we showed generically that it is possible to solve the query release problem using a small number of black- 
box calls to a learning algorithm, without accessing the database except through the learning algorithm. 
This has interesting implications for any class of algorithms that may make only restricted access to the 
database. For example, this also proves that if it is possible to agnostically learn some concept class C while 
preserving e-differential privacy (even using algorithms that do not fit into the SQ model), then it is possible 
to release the same class while preserving Ts « log |X|e-differential privacy. 
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